Data linkage system and processing monitoring system

ABSTRACT

A data linkage system including a data collection system that collects data held by an information system and a data storage system that stores data held by a plurality of the information systems and collected by the data collection system includes a processing monitoring system that monitors processing at each stage on data in the data linkage system, and when processing which failed is detected, the processing monitoring system has this processing re-executed.

INCORPORATION BY REFERENCE

This application is based upon, and claims the benefit of priority from,corresponding Japanese Patent Application No. 2020-034412 filed in theJapan Patent Office on Feb. 28, 2020, the entire contents of which areincorporated herein by reference.

BACKGROUND Field of the Invention

The present disclosure relates to a data linkage system and a processingmonitoring system that collect and store data held by a plurality ofinformation systems.

Description of Related Art

Conventionally, an information processing system that links with aplurality of external systems is known.

SUMMARY

The data linkage system of the present disclosure is a data linkagesystem including a data collection system that collects data held by aninformation system and a data storage system that stores the data heldby a plurality of the information systems and collected by the datacollection system and includes a processing monitoring system thatmonitors processing at each stage on the data in the data linkagesystem, and the processing monitoring system is characterized in that,when the processing which failed is detected, the processing monitoringsystem has this processing re-executed.

The processing monitoring system of the present disclosure is aprocessing monitoring system that monitors processing at each stage onthe data in the data linkage system including the data collection systemthat collects data held by the information system and the data storagesystem that stores the data held by a plurality of the informationsystems and collected by the data collection system, and the processingmonitoring system is characterized in that, when the processing whichfailed is detected, the processing monitoring system has this processingre-executed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system according to one embodiment of thepresent disclosure;

FIG. 2 is a block diagram of a pipeline included in the data storagesystem shown in FIG. 1;

FIG. 3 is a block diagram of a pipeline orchestrator shown in FIG. 1;

FIG. 4 is a diagram showing an example of an operation flow of thesystem shown in FIG. 1 when data held by an information system iscollected by a POST connector and transmitted to the pipeline;

FIG. 5 is a flowchart of the operation of the POST connector shown inFIG. 4 when a file is transmitted to the pipeline;

FIG. 6 is a diagram showing an example of the operation flow of thesystem shown in FIG. 1 when the data held by the information system iscollected by a GET connector and passed to the pipeline;

FIG. 7 is a diagram showing an example of the operation flow of thesystem shown in FIG. 1 when the data held by the information system iscollected by a POST agent and transmitted to the pipeline;

FIG. 8 is a flowchart of the operation of the POST agent shown in FIG. 7when a file is transmitted to the pipeline;

FIG. 9 is a diagram showing an example of the operation flow of thesystem shown in FIG. 1 when the data held by the information system iscollected by a GET agent and passed to the pipeline;

FIG. 10 is a sequence diagram of a part of the operation of the datalinkage system shown in FIG. 1 when the data storage system stores data;

FIG. 11 is a sequence diagram of operations following the operationsshown in FIG. 10;

FIG. 12 is a flowchart of the operation of a masking processing unit inmasking processing shown in FIG. 10;

FIG. 13 is a diagram showing an example of a data management table usedin the operation shown in FIG. 12;

FIG. 14 is a sequence diagram of the operation of the data linkagesystem shown in FIG. 1 when the masking processing unit fails to processthe data;

FIG. 15 is a sequence diagram of the operation of the data linkagesystem shown in FIG. 1 when an application unit requests update of thedata of a specific information system stored in the data storage system;and

FIG. 16 is a flowchart of the operation of the data linkage system shownin FIG. 1 when its own configuration is changed in response to a changein the configuration of a specific information system.

DETAILED DESCRIPTION

An embodiment of the present disclosure will be described below usingthe accompanying drawings.

First, configuration of a system according to the embodiment of thepresent disclosure will be explained.

FIG. 1 is a block diagram of a system 10 according to the presentembodiment.

As shown in FIG. 1, the system 10 includes a data source unit 20 thatproduces data and a data linkage system 30 that links the data generatedby the data source unit 20.

The data source unit 20 includes an information system 21 that producesdata. The information system 21 includes a configuration managementserver 21 a that stores the configuration and settings of theinformation system 21. The data source unit 20 may include at least oneinformation system in addition to the information system 21. Examples ofthe information system are IoT (Internet of Things) systems such asremote management systems that remotely manage image forming apparatusessuch as MFP (Multifunction Peripheral) and printers and in-house systemssuch as ERP (Enterprise Resource Planning) and production managementsystems. Each of the information systems may be configured by onecomputer or may be configured by a plurality of computers. Theinformation system may hold a file of structured data. The informationsystem may hold a file of unstructured data. The information system mayhold a database of structured data.

The data source unit 20 includes a POST connector 22 as the datacollection system that acquires a file of structured data orunstructured data held by the information system and transmits theacquired file to a pipeline which will be described later of the datalinkage system 30. The data source unit 20 may include at least one POSTconnector having the same configuration as the POST connector 22 inaddition to the POST connector 22. The POST connector may be configuredby a computer that constitutes an information system in which the POSTconnector itself acquires files. The POST connector is alsoconfiguration of the data linkage system 30.

The data source unit 20 includes a POST agent 23 as the data collectionsystem that acquires structured data from a database of the structureddata held by the information system and transmits the acquiredstructured data to a pipeline which will be described later of the datalinkage system 30. The data source unit 20 may include at least one POSTagent having the same configuration as the POST agent 23 in addition tothe POST agent 23. The POST agent may be configured by a computer thatconstitutes an information system in which the POST agent itselfacquires structured data. The POST agent is also configuration of thedata linkage system 30.

The data source unit 20 includes a GET agent 24 as the data collectionsystem that generates structured data for linkage on the basis of thedata held by the information system. The data source unit 20 may includeat least one GET agent having the same configuration as the GET agent 24in addition to the GET agent 24. The GET agent may be configured by acomputer that constitutes an information system that holds the data thatis a source of generation of the structured data for linkage. The GETagent is also configuration of the data linkage system 30.

The data linkage system 30 includes a data storage system 40 that storesdata generated by the data source unit 20, an application unit 50 thatuses the data stored in the data storage system 40, and a controlservice unit 60 that executes various controls on the data storagesystem 40 and the application unit 50.

The data storage system 40 includes a pipeline 41 that stores the datagenerated by the data source unit 20. The data storage system 40 mayinclude at least one pipeline in addition to the pipeline 41. Since thedata configuration in the information system may be different for eachinformation system, the data storage system 40 basically includes apipeline for each information system. Each of the pipelines may beconfigured by one computer or may be configured by a plurality ofcomputers.

FIG. 2 is a block diagram of a pipeline 70 included in the data storagesystem 40.

As shown in FIG. 2, the pipeline 70 includes a primary storage 71 havinga storage area for storing data received from the POST connector, thePOST agent, the GET connector which will be described later, or a GETagent which will be described later, a masking processing unit 72 as thedata conversion system that executes masking processing as dataconversion processing for data related to privacy such as personalinformation of a user of the information system in the data stored inthe primary storage 71, a data transfer processing unit 73 that executesdata transfer processing for transferring data for which the maskingprocessing has been executed by the masking processing unit 72 to a bigdata analysis unit 44 (see FIG. 1) which will be described later, and asecondary storage 74 having a storage area for storing data to betransferred to the big data analysis unit 44. The reason why the primarystorage 71 is provided is that in the data processing, if the processingfails in a process after the process of storing the data in the primarystorage 71 such as processes of masking processing and a data transferprocessing, re-execution of the failed processing using the data storedin the primary storage 71 is made possible without retransmitting thedata from the data source unit 20 to the data linkage system 30, whichhas a high network communication cost. The primary storage 71 and thesecondary storage 74 are not merely storage devices but are systemscapable of executing various types of processing which will be describedlater.

As shown in FIG. 1, the data storage system 40 includes a GET connector42 as the data collection system that acquires a file of structured dataor unstructured data held by the information system and links theacquired file to the pipeline. The data storage system 40 may include atleast one GET connector having the same configuration as the GETconnector 42 in addition to the GET connector 42. The GET connector maybe configured by a computer that constitutes a pipeline in which the GETconnector itself links files.

The system 10 includes a POST connector in the data source unit 20 foran information system that does not support the acquisition ofstructured data or unstructured data files from the data storage system40 side. On the other hand, the system 10 includes the GET connector inthe data storage system 40 for an information system that supports theacquisition of a file of structured data or unstructured data from thedata storage system 40 side.

The data storage system 40 includes a GET agent 43 as a data collectionsystem that acquires structured data generated by the GET agent andlinks the acquired structured data to a pipeline. The data storagesystem 40 may include at least one GET agent having the sameconfiguration as the GET agent 43 in addition to the GET agent 43. TheGET agent may be configured by a computer that constitutes a pipeline inwhich the GET agent itself links structured data.

The system 10 includes a POST agent in the data source unit 20 for aninformation system that does not support the acquisition of structureddata from the data storage system 40 side. On the other hand, the system10 includes a GET agent in the data source unit 20 and a GET agent inthe data storage system 40 for an information system that supports theacquisition of structured data from the data storage system 40 side.

The data storage system 40 includes a big data analysis unit 44 as adata conversion system that executes final conversion processing as dataconversion processing for converting data stored by a plurality ofpipelines into a form that can be searched or aggregated in a querylanguage such as a database language such as SQL. The big data analysisunit 44 can also execute a search or aggregation in response to a searchrequest or an aggregation request from the application unit 50 side onthe data for which the final conversion processing has been executed.The big data analysis unit 44 may be configured by one computer or maybe configured by a plurality of computers.

The final conversion processing may include data integration processingfor integrating data of a plurality of information systems as dataconversion processing. When the system 10 includes a remote managementsystem located in Asia to remotely manage a large number of imageforming apparatuses located in Asia, a remote management system locatedin Europe to remotely manage a large number of image forming apparatuseslocated in Europe, and a remote management system located in the UnitedStates to remotely manage a large number of image forming apparatuseslocated in the United States as information systems, each of these threeremote management systems includes a device management table thatmanages an image forming apparatus managed by the remote managementsystem itself. The device management table is information indicatingvarious types of information of the image forming apparatus inassociation with an ID assigned to each image forming apparatus. Here,since each of the three remote management systems has its own devicemanagement table, there is a possibility that the same ID is assigned todifferent image forming apparatuses among the device management tablesof the three remote management systems. Therefore, when the big dataanalysis unit 44 integrates the device management tables of the threeremote management systems to generate one device management table, theID of the image forming apparatus is reassigned so as not to causeduplication.

The application unit 50 includes an application service 51 that executesa specific operation instructed by a user such as data display or dataanalysis by using the data managed by the big data analysis unit 44. Theapplication unit 50 may include at least one application service inaddition to the application service 51. Each of the application servicesmay be configured by one computer or may be configured by a plurality ofcomputers.

The application unit 50 includes an API platform 52 that provides an API(Application Program Interface) that executes a specific operation byusing the data managed by the big data analysis unit 44. The APIplatform 52 may be configured by one computer or may be configured by aplurality of computers. For example, as the API provided by the APIplatform 52, there are an API that transmits data of a remaining amountof consumables collected by the remote management system from the imageforming apparatus to a consumables ordering system outside of the system10, that orders consumables when the remaining amount of consumablessuch as toner of the image forming apparatus is equal to or less than aspecific amount and an API that transmits various types of datacollected by the remote management system from the image formingapparatus to a failure prediction system outside of the system 10, thatpredicts the failure of the image forming apparatus.

The control service unit 60 includes a pipeline orchestrator 61 as aprocessing monitoring system that monitors the processing of each stageof data in the data source unit 20, the data storage system 40, and theapplication unit 50. Each of the pipeline orchestrators 61 may beconfigured by one computer or may be configured by a plurality ofcomputers.

FIG. 3 is a block diagram of the pipeline orchestrator 61.

As shown in FIG. 3, the pipeline orchestrator 61 includes a triggerprocessing unit 81 that processes a trigger of an operation of thepipeline orchestrator 61, an action description unit 82 that stores aplurality of operation scenarios of the pipeline orchestrator 61, and anaction processing unit 83 that executes the operation of the pipelineorchestrator 61.

As shown in FIG. 1, the control service unit 60 includes a configurationmanagement server 62 that stores configuration and settings of the datastorage system 40 and automatically executes deployment as necessary.The configuration management server 62 may be configured by one computeror may be configured by a plurality of computers. The configurationmanagement server 62 configures a configuration change system thatchanges the configuration of the data linkage system 30.

The control service unit 60 includes a configuration management gateway63 connected to the configuration management server of the informationsystem and collects information for detecting a change in theconfiguration of the database or unstructured data in the informationsystem, that is, a change in the configuration of the data in theinformation system. The configuration management gateway 63 may beconfigured by one computer or may be configured by a plurality ofcomputers.

The control service unit 60 includes a key management service 64 thatencrypts and stores security information such as key information andconnection character strings required for linking each system such as aninformation system. The key management service 64 may be configured byone computer or may be configured by a plurality of computers.

The control service unit 60 includes a management API 65 that receivesrequests from the data storage system 40 and the application unit 50.The management API 65 may be configured by one computer or may beconfigured by a plurality of computers.

The control service unit 60 includes an authentication/authorizationservice 66 that executes authentication/authorization of the applicationservice of the application unit 50. The authentication/authorizationservice 66 may be configured by one computer or may be configured by aplurality of computers. The authentication/authorization service 66 canconfirm, for example, whether or not the application service ispermitted to request the update of the data of the information systemstored in the data storage system 40.

Next, the operation of the system 10 will be described.

First, the operation of the system 10 when the data held by theinformation system 21 is collected by the POST connector 22 andtransmitted to the pipeline 41 will be described.

FIG. 4 is a diagram showing an example of an operation flow of thesystem 10 when the data held by the information system 21 is collectedby the POST connector 22 and transmitted to the pipeline 41.

In the example shown in FIG. 4, the information system 21 is aproduction management system 100.

As shown in FIG. 4, the production management system 100 includes aproduction management server 101 that executes production management anda storage 102 that stores a file of structured data or unstructureddata.

The production management server 101 executes backup for storingstructured data or unstructured data files in the storage 102 by batchprocessing (S201).

After the processing at S201, the production management server 101instructs the POST connector 22 to transfer the file stored in thestorage 102 at S201 to the pipeline (S202). Here, the productionmanagement server 101 includes identification information of the filestored in the storage 102 at S201 in the instruction at S202.

Upon receipt of the instruction at S202, the POST connector 22 acquiresthe file specified by the identification information included in theinstruction at S202 from the storage 102 (S203).

After the processing at S203, the POST connector 22 transmits the fileacquired at S203 to the pipeline 41 with which the POST connector 22itself is associated (S204).

FIG. 5 is a flowchart of the operation of the POST connector 22 when afile is transmitted to the pipeline 41.

As shown in FIG. 5, the POST connector 22 assigns a transaction ID asidentification information to the current transaction for transmitting afile to the pipeline 41 (S221). Here, the transaction ID is, forexample, a numerical value and is incremented each time a newtransaction occurs in the POST connector 22.

The POST connector 22 determines whether or not the data targeted forthe current transaction is larger than a specific unit of processing(S222). Here, the specific unit of processing is, for example, aspecific number of files.

When the POST connector 22 determines at S222 that the data targeted forthe current transaction is larger than the specific unit of processing,the POST connector 22 divides the data targeted for the currenttransaction into specific units of processing (S223).

When the POST connector 22 determines at S222 that the data targeted forthe current transaction is equal to or smaller than the specific unit ofprocessing, or when the processing at S223 is finished, the POSTconnector 22 assigns the processing ID as identification information toeach data in the unit of processing (S224). Here, the processing ID is,for example, a numerical value and is incremented each time new data ofa specific unit of processing is generated in the POST connector 22.

After the processing at S224, the POST connector 22 starts transmittingthe data targeted for the current transaction to the pipeline 41 foreach unit of processing (S225).

Next, the POST connector 22 determines whether or not the number offiles transmitted to the pipeline 41 per specific unit time has exceededthe specific number (S226).

When the POST connector 22 determines at S226 that the number of filestransmitted to the pipeline 41 per specific unit time does not exceedthe specific number, the POST connector 22 determines whether or not thetransmission of the data targeted for the current transaction to thepipeline 41 has been completed (S227).

When the POST connector 22 determines at S227 that the transmission ofthe data targeted for the current transaction to the pipeline 41 has notbeen completed, the POST connector 22 executes the processing at S226.

When the POST connector 22 determines at S227 that the transmission ofthe data targeted for the current transaction to the pipeline 41 hasbeen completed, the POST connector 22 ends the operation shown in FIG.5.

When the POST connector 22 determines at S226 that the number of filestransmitted to the pipeline 41 per specific unit time has exceeded thespecific number, the POST connector 22 instructs scale-out of thepipeline 41 and start of parallel processing by the pipeline 41 to thepipeline orchestrator 61 (S228). Therefore, the pipeline orchestrator 61scales out the pipeline 41 to a specific state in accordance with theinstruction at S228 and instructs the pipeline 41 to start parallelprocessing.

Next, the POST connector 22 determines whether or not the transmissionof the data targeted for the current transaction to the pipeline 41 hasbeen completed until it determines that the transaction of the datatargeted for the current transaction to the pipeline 41 has beencompleted (S229).

When the POST connector 22 determines at S229 that transmission of thedata targeted for the current transaction to the pipeline 41 has beencompleted, the POST connector 22 instructs the scale-in of the pipeline41 and the end of parallel processing by the pipeline 41 to the pipelineorchestrator 61 (S230). Therefore, the pipeline orchestrator 61 scalesin the pipeline 41 to the original state in accordance with theinstruction at S230 and instructs the pipeline 41 to end the parallelprocessing.

The POST connector 22 ends the operation shown in FIG. 5 after theprocessing at S230.

Next, the operation of the system 10 when the data held by theinformation system is collected by the GET connector 42 and passed tothe pipeline will be described.

FIG. 6 is a diagram showing an example of the operation flow of thesystem 10 when the data held by the information system is collected bythe GET connector 42 and passed to the pipeline.

In the example shown in FIG. 6, the information system is the remotemanagement system 120 of the image forming apparatus. The example shownin FIG. 6 is an example of an operation when the user instructs theremote management system 120 to acquire a maintenance report includingsensor information including output values of various sensors of theimage forming apparatus.

As shown in FIG. 6, the remote management system 120 includes a usercommunication server 121 that receives instructions from users, aback-end processing server 122 that executes processing in response toinstructions from users, a command server 123 that transmits variouscommands to the image forming apparatus, a device communication server124 that receives data from the image forming apparatus, the database125 that stores various types of data of the image forming apparatus tobe managed by the remote management system 120, and a storage 126 thatstores the files of structured data or unstructured data. The remotemanagement system 120 manages a large number of image formingapparatuses including the image forming apparatus 130. The database 125stores the device ID as the identification information of the imageforming apparatus for the image forming apparatus to be managed by theremote management system 120.

The user of the remote management system 120 can transmit an instructionto acquire the maintenance report of the image forming apparatus 130 tothe remote management system 120. This instruction includes the deviceID of the image forming apparatus 130 from which the maintenance reportis acquired. When the user communication server 121 of the remotemanagement system 120 receives the instruction to acquire themaintenance report, the user communication server 121 transmits thereceived instruction to the back-end processing server 122 (S251).

When the back-end processing server 122 receives the instruction toacquire the maintenance report transmitted by the user communicationserver 121 at S251, the back-end processing server 122 transmits arequest for transmission of the maintenance report acquisition commandfor acquiring the maintenance report to the command server 123 (S252).This request includes the device ID that was included in the instructionto acquire the maintenance report.

When the command server 123 receives the request for transmission of themaintenance report acquisition command transmitted by the back-endprocessing server 122 at S252, the command server 123 transmits themaintenance report acquisition command to the image forming apparatus130 specified by the device ID included in the request (S253).

When the image forming apparatus 130 receives the maintenance reportacquisition command transmitted by the command server 123 at S253, theimage forming apparatus 130 transmits the maintenance report of theimage forming apparatus 130 itself to the remote management system 120(S254). Here, the image forming apparatus 130 includes the device ID ofthe image forming apparatus 130 itself in the maintenance report.

When the device communication server 124 of the remote management system120 receives the maintenance report transmitted by the image formingapparatus 130 at S254, the device communication server 124 determineswhether or not the device ID included in the received maintenance reportis included in the database 125. (S255).

When the device communication server 124 determines at S255 that thedevice ID included in the received maintenance report is included in thedatabase 125, the device communication server 124 stores the receivedmaintenance report in the storage 126 (S256).

The GET connector 42 of the data linkage system 30 periodically searchesthe storage 126 of the remote management system 120, which is aninformation system with which the GET connector 42 itself is associated,with respect to the maintenance report file of the specific imageforming apparatus (S257).

When the GET connector 42 confirms that the maintenance report file ofthe specific image forming apparatus 130 exists in the storage 126, theGET connector 42 acquires this file from the storage 126 (S258).

After the processing at S258, the GET connector 42 passes the fileacquired at S258 to the pipeline with which the GET connector 42 itselfis associated (S259).

When passing a file to the pipeline, the GET connector 42 executes anoperation similar to the operation shown in FIG. 5. That is, the GETconnector 42 assigns a transaction ID to the current transaction.Further, the GET connector 42 divides the target data of the currenttransaction into specific units of processing when the target data ofthe current transaction is larger than the specific units of processing.Further, the GET connector 42 assigns a processing ID to each processingunit of data. In addition, when the number of files passed to thepipeline per specific unit of time has exceeded the specific number, theGET connector 42 instructs the scale-out of the pipeline and the startof parallel processing by the pipeline to the pipeline orchestrator 61and then, when passing of the data targeted for the current transactionto the pipeline is completed, the GET connector 42 instructs thescale-in of the pipeline and the end of parallel processing by thepipeline to the pipeline orchestrator 61.

Next, the operation of the system 10 when the data held by theinformation system is collected by the POST agent 23 and transmitted tothe pipeline will be described.

FIG. 7 is a diagram showing an example of the operation flow of thesystem 10 when the data held by the information system is collected bythe POST agent 23 and transmitted to the pipeline.

In the example shown in FIG. 7, the information system is the remotemanagement system 120 of the image forming apparatus similarly to theexample shown in FIG. 6. The database 125 stores event informationindicating an event that has occurred in the image forming apparatusmanaged by the remote management system 120. The example shown in FIG. 7is an example of the operation of the system 10 when the image formingapparatus 130 managed by the remote management system 120 transmitsevent information indicating an event generated in the image formingapparatus 130 itself to the remote management system 120.

When an event such as an error occurs in the image forming apparatus 130itself, the image forming apparatus 130 transmits event informationindicating the event occurring in the image forming apparatus 130 itselfto the device communication server 124 of the remote management system120 (S271). For example, as an error that occurs in the image formingapparatus 130, there are a paper jam indicating that paper is jammedinside the image forming apparatus 130 and a cover open indicating thatthe cover of the image forming apparatus 130 is in the open state.

When the device communication server 124 of the remote management system120 receives the event information transmitted by the image formingapparatus 130 at S271, the device communication server 124 updates thedatabase 125 with the received event information (S272).

The POST agent 23 confirms at a specific timing whether or not the eventinformation stored in the database 125 has been changed (S273). Theconfirmation at S273 may be executed, for example, at the time ofperiodic backup of the database 125, may be executed when the database125 itself detects a change in the database 125, or may be executed whenthe API for change of the database 125 is called in the remotemanagement system 120.

When the POST agent 23 detects a change in the event information in thedatabase 125 as a result of the confirmation at S273, the POST agent 23acquires data indicating the content of the change in the eventinformation from the database 125 (S274).

After the processing at S274, the POST agent 23 transmits the dataacquired at S274 to the pipeline of the data linkage system 30 withwhich the POST agent 23 itself is associated (S275).

FIG. 8 is a flowchart of the operation of the POST agent 23 when a fileis transmitted to the pipeline.

As shown in FIG. 8, the POST agent 23 assigns a transaction ID to thecurrent transaction that transmits a file to the pipeline (S291). Here,the transaction ID is, for example, a numerical value and is incrementedeach time a new transaction occurs in the POST agent 23.

The POST agent 23 determines whether or not the data targeted for thecurrent transaction is larger than a specific unit of processing (S292).Here, the specific unit of processing is, for example, a specific numberof tables.

When the POST agent 23 determines at S292 that the data targeted for thecurrent transaction is larger than the specific unit of processing, thePOST agent 23 divides the data targeted for the current transaction intospecific units of processing (S293).

When the POST agent 23 determines at S292 that the data targeted for thecurrent transaction is equal to or smaller than a specific unit ofprocessing, or when the processing at S293 is finished, the POST agent23 assigns the processing ID as identification information to each dataof the unit of processing (S294). Here, the processing ID is, forexample, a numerical value, and is incremented each time data of aspecific unit of processing newly occurs in the POST agent 23 in thesame transaction.

After the processing at S294, the POST agent 23 starts transmission ofthe data targeted for the current transaction to the pipeline for eachunit of processing (S295).

Next, the POST agent 23 determines whether or not the amount of datatransmitted to the pipeline per specific unit of time has exceeded thespecific amount (S296).

When the POST agent 23 determines at S296 that the amount of datatransmitted to the pipeline per specific unit of time does not exceedthe specific amount, the POST agent 23 determines whether or nottransmission of the data targeted for the current transaction to thepipeline has been completed (S297).

When the POST agent 23 determines at S297 that the transmission of thedata targeted for the current transaction to the pipeline has not beencompleted, the POST agent 23 executes the processing at S296.

When the POST agent 23 determines at S297 that the transmission of thedata targeted for the current transaction to the pipeline has beencompleted, the POST agent 23 ends the operation shown in FIG. 8.

When the POST agent 23 determines at S296 that the amount of datatransmitted to the pipeline per specific unit of time has exceeded thespecific amount, the POST agent 23 instructs scale-out of the pipelineand start of parallel processing by the pipeline to the pipelineorchestrator 61 (S298). Therefore, the pipeline orchestrator 61 scalesout the pipeline to a specific state in accordance with the instructionat S298 and instructs the pipeline to start parallel processing.

Then, the POST agent 23 determines whether or not transmission of thedata targeted for the current transaction to the pipeline has beencompleted until the POST agent 23 determines that the transmission ofthe data targeted for the current transaction to the pipeline has beencompleted (S299).

When the POST agent 23 determines at S299 that the transmission of thedata targeted for the current transaction to the pipeline has beencompleted, the POST agent 23 instructs the scale-in of the pipeline andthe end of parallel processing by the pipeline to the pipelineorchestrator 61 (S300). Therefore, the pipeline orchestrator 61 scalesin the pipeline to the original state in accordance with the instructionat S300 and instructs the pipeline to end the parallel processing.

The POST agent 23 ends the operation shown in FIG. 8 after theprocessing at S300.

Next, the operation of the system 10 when the data held by theinformation system is collected by the GET agent 43 and passed to thepipeline will be described.

FIG. 9 is a diagram showing an example of the operation flow of thesystem 10 when the data held by the information system is collected bythe GET agent 43 and passed to the pipeline.

In the example shown in FIG. 9, the information system is the productionmanagement system 100 similarly to the example shown in FIG. 4.

As shown in FIG. 9, the GET agent 24 of the production management system100 generates structured data for linkage at a specific timing on thebasis of the data stored in the storage 102 (S321).

The GET agent 43 of the data linkage system 30 periodically inquires theGET agent 24 of the production management system 100, which is aninformation system with which the GET agent 43 itself is associated, forpresence or absence of structured data for linkage (S322).

When the GET agent 43 confirms that the structured data for linkageexists in the GET agent 24, the GET agent 43 acquires the structureddata from the GET agent 24 (S323).

After the processing at S323, the GET agent 43 passes the structureddata acquired at S323 to the pipeline with which the GET agent 43 itselfis associated (S324).

When a file is to be passed to the pipeline, the GET agent 43 executesan operation similar to the operation shown in FIG. 8. That is, the GETagent 43 assigns a transaction ID to the current transaction. Further,the GET agent 43 divides the data targeted for the current transactioninto specific units of processing when the data targeted for the currenttransaction is larger than the specific unit of processing. Further, theGET agent 43 assigns a processing ID to each unit of processing of thedata. In addition, when the amount of data passed to the pipeline perspecific unit of time has exceeded the specific amount, the GET agent 43instructs the scale-out of the pipeline and the start of parallelprocessing by the pipeline to the pipeline orchestrator 61 and then,when passing of the data targeted for the current transaction to thepipeline has been completed, the GET agent 43 instructs the scale-in ofthe pipeline and the end of parallel processing by the pipeline to thepipeline orchestrator 61.

Next, the operation of the data linkage system 30 when the data storagesystem 40 stores data will be described.

FIG. 10 is a sequence diagram of a part of the operation of the datalinkage system 30 when the data storage system 40 stores data.

As shown in FIG. 10, when the primary storage 71 of the pipeline 70receives the data of a specific unit of processing from the datacollection system, that is, the POST connector, POST agent, GETconnector or GET agent, it stores the received data (S341). Next, theprimary storage 71 notifies the pipeline orchestrator 61 of an eventindicating the completion of data storage (S342).

When the trigger processing unit 81 of the pipeline orchestrator 61receives the event notified by the primary storage 71 at S342, thetrigger processing unit 81 analyzes the content of this event, calls ascenario corresponding to this event, that is, a scenario of the maskingprocessing from the action description unit 82 (S343), and notifies thescenario called at S343 to the action processing unit 83 (S344).Therefore, the action processing unit 83 instructs the maskingprocessing unit 72 of the pipeline 70 to execute the processing based onthe scenario notified at S344, that is, to execute the maskingprocessing on the data stored in the primary storage 71 at S341 (S345).

Upon receipt of the instruction at S345, the masking processing unit 72executes the masking processing on the data stored in the primarystorage 71 at S341. That is, the masking processing unit 72 firstacquires the data stored in the primary storage 71 at S341 from theprimary storage 71 (S346). Next, the masking processing unit 72 executesthe masking processing on the data acquired at S346 (S347). Next, themasking processing unit 72 passes the data for which the maskingprocessing was executed at S347 to the data transfer processing unit 73(S348). Then, the masking processing unit 72 notifies the pipelineorchestrator 61 of an event indicating completion of the maskingprocessing (S349).

When the trigger processing unit 81 of the pipeline orchestrator 61receives the event notified by the masking processing unit 72 at S349,the trigger processing unit 81 analyzes the content of this event, callsa scenario corresponding to this event, that is, a scenario of the datatransfer processing from the action description unit 82 (S350), andnotifies the scenario called at S350 to the action processing unit 83(S351). Therefore, the action processing unit 83 instructs the datatransfer processing unit 73 of the pipeline 70 to execute the processingbased on the scenario notified at S351, that is, to execute the datatransfer processing on the data for which the masking processing wasexecuted at S347 (S352).

FIG. 11 is a sequence diagram of operations following the operationsshown in FIG. 10.

As shown in FIG. 11, when the data transfer processing unit 73 receivesthe instruction at S352, the data transfer processing unit 73 executesthe data transfer processing on the data for which the maskingprocessing has been executed by the masking processing unit 72. That is,the data transfer processing unit 73 first stores the data passed fromthe masking processing unit 72 at S348 as data for transfer to the bigdata analysis unit 44 in the secondary storage 74 (S353). Next, the datatransfer processing unit 73 transfers the data stored in the secondarystorage 74 at S353 to the big data analysis unit 44 via the secondarystorage 74 (S354). Then, the data transfer processing unit 73 notifiesthe pipeline orchestrator 61 of an event indicating the completion ofthe data transfer processing (S355).

When the trigger processing unit 81 of the pipeline orchestrator 61receives the event notified by the data transfer processing unit 73 atS355, the trigger processing unit 81 analyzes the content of this event,calls a scenario corresponding to this event, that is, a scenario offinal conversion processing from the action description unit 82 (S356),and notifies the scenario called at S356 to the action processing unit83 (S357). Therefore, the action processing unit 83 instructs the bigdata analysis unit 44 to execute the processing based on the scenarionotified at S357, that is, to execute the final conversion processingfor the data stored in the secondary storage 74 at S354 (S358).

Upon receipt of the instruction at S358, the big data analysis unit 44executes the final conversion processing on the data transferred by thedata transfer processing unit 73. That is, the big data analysis unit 44first converts the data transferred from the data transfer processingunit 73 at S354 into a form that can be searched and aggregated in aspecific query language (S359). Then, the big data analysis unit 44notifies the pipeline orchestrator 61 of an event indicating thecompletion of the final conversion processing (S360).

Next, the operation of the masking processing unit 72 in the maskingprocessing at S347 will be described.

FIG. 12 is a flowchart of the operation of the masking processing unit72 in the masking processing.

The masking processing unit 72 executes the operation shown in FIG. 12for each unit of processing of the data.

As shown in FIG. 12, the masking processing unit 72 writes informationindicating that the masking processing is being executed for the data tobe masked this time in a data management table 90 (see FIG. 13) as datamanagement information for managing history of the data processing to belinked (S381).

FIG. 13 is a diagram showing an example of the data management table 90used in the operation shown in FIG. 12.

The data management table 90 shown in FIG. 13 includes a transaction ID,a processing ID, a storage type indicating a storage in which dataidentified by combination of the transaction ID and the processing ID isstored, a storage name indicating the name of the file when the dataidentified by the combination of the transaction ID and the processingID is stored in the storage, the last update date and time indicatingthe date and time when the information was stored in the data managementtable 90, a processing name indicating the name of the processing forthe data identified by the combination of the transaction ID and theprocessing ID, and a processing state indicating the state of theprocessing indicated by the processing name.

There are a primary storage and a secondary storage in the storage type.

The processing name includes Masking indicating the masking processingand Transfer indicating the data transfer processing. At S381, Maskingis written.

In the processing state, there are Processing indicating that theprocessing indicated by the processing name is being executed, Completedindicating that the processing indicated by the processing name has beencompleted normally, and Error indicating that the processing indicatedby the processing name has failed. At S381, Processing is written.

As shown in FIG. 12, the masking processing unit 72 starts the maskingprocessing on the target data after the processing at S381 (S382).

Next, the masking processing unit 72 determines whether or not thefailure of the masking processing started at S382, that is, the failureof data conversion has been detected (S383).

When the masking processing unit 72 determines at S383 that the failureof the masking processing has not been detected, it determines whetheror not the masking processing started at S382 has been completed (S384).

When the masking processing unit 72 determines at S384 that the maskingprocessing has not been completed, the masking processing unit 72executes the processing at S383.

When the masking processing unit 72 determines at S383 that it hasdetected the failure of the masking processing, it notifies the pipelineorchestrator 61 of an event indicating the failure of the maskingprocessing (S385). This event includes the transaction ID and processingID of the target data.

Next, the masking processing unit 72 writes information indicating thatthe masking processing has failed with respect to the data to be maskedthis time in the data management table 90 (S386), and ends the operationshown in FIG. 12. The “processing name” and the “processing state” inthe information written at S386 are “Masking” and “Error”, respectively.

When the masking processing unit 72 determines at S384 that the maskingprocessing has been completed, the masking processing unit 72 writesinformation indicating that the masking processing has been normallycompleted for the data to be masked this time in the data managementtable 90 (S387) and ends the operation shown in FIG. 12. The “processingname” and “processing state” in the information written at S387 are“Masking” and “Completed”, respectively.

Although the operation of the masking processing unit 72 in the maskingprocessing at S347 has been described above, the same applies to theoperation of the data transfer processing unit 73 in the data transferprocessing at S354 and the operation of the big data analysis unit 44 inthe final conversion processing at S359.

Next, the operation of the data linkage system 30 when the maskingprocessing unit 72 fails to process the data will be described.

FIG. 14 is a sequence diagram of the operation of the data linkagesystem 30 when the masking processing unit 72 fails to process the data.

If the masking processing fails during the execution of the operationshown in FIG. 10, the masking processing unit 72 notifies the pipelineorchestrator 61 of an event indicating the failure of the maskingprocessing as shown in FIG. 14 (S401). The notification at S401corresponds to the notification at S385 (see FIG. 12).

When the trigger processing unit 81 of the pipeline orchestrator 61receives the event notified by the masking processing unit 72 at S401,the trigger processing unit 81 analyzes the content of this event andcalls a scenario corresponding to this event, that is, the scenario ofre-execution of the masking processing from the action description unit82 (S402) and notifies the scenario called at S402 to the actionprocessing unit 83 (S403). Therefore, the action processing unit 83instructs the masking processing unit 72 of the pipeline 70 to executethe processing based on the scenario notified at S403, that is, toexecute the masking processing on the data stored in the primary storage71 at S341 (S404). Here, the action processing unit 83 specifies theinformation whose final update date and time is the latest in theinformation included in the data management table 90 for the dataspecified by the combination of the transaction ID and the processing IDincluded in the event notified by the masking processing unit 72 atS401, and when the processing state in the specified information is notCompleted, that is, Processing or Error, the action processing unit 83instructs execution of the masking processing for this data to themasking processing unit 72 of the pipeline 70.

After the processing at S404, the processing after the processing atS346 shown in FIG. 10 is executed.

In the above, the operation of the data linkage system 30 when themasking processing unit 72 fails to process the data has been described,but even when configuration other than the masking processing unit 72 inthe data storage system 40 such as the data transfer processing unit 73and the big data analysis unit 44 fails to process data, or whenconfiguration other than the data storage system 40 in the data linkagesystem 30 such as a data collection system fails to process data, thedata linkage system 30 can re-execute the processing by the samemechanism.

The data stored in the primary storage 71 is not frequently used.Therefore, the primary storage 71 may move the data for which a specificperiod has passed since it was stored in the primary storage 71 itselfto a specific storage area outside the pipeline. When the primarystorage 71 moves the data to a specific storage area outside thepipeline, the primary storage 71 may compress the data and then, movethe data. The primary storage 71 moves the data to a specific storagearea outside the pipeline, and then, notifies the combination of thetransaction ID and the processing ID of the data having been moved tothe specific storage area outside the pipeline to the pipelineorchestrator 61. When the pipeline orchestrator 61 instructs the maskingprocessing unit 72 of the pipeline 70 to execute the masking processingon the data having been moved to a specific storage area outside thepipeline, the pipeline orchestrator 61 instructs the primary storage 71to restore this data to the primary storage 71. Therefore, the primarystorage 71 acquires the data specified by the pipeline orchestrator 61from a specific storage area outside the pipeline and stores it in theprimary storage 71 itself. Here, when the data specified by the pipelineorchestrator 61 is compressed, the primary storage 71 decompresses thisdata and then, stores the data in the primary storage 71 itself.

In the above, the data stored in the primary storage 71 has beendescribed, but the same applies to the data stored in the secondarystorage 74. That is, the secondary storage 74 may move the data forwhich a specific period has passed since it was stored in the secondarystorage 74 itself to a specific storage area outside the pipeline andrestores the data having been moved to the specific storage area outsidethe pipeline to the secondary storage 74 itself in accordance with theinstruction of the pipeline orchestrator 61. When the secondary storage74 moves the data to a specific storage area outside the pipeline, thesecondary storage 74 may compress the data and then, move the data.

Next, the operation of the data linkage system 30 when the applicationunit 50 requests the update of the data of the specific informationsystem stored in the data storage system 40 will be described.

FIG. 15 is a sequence diagram of the operation of the data linkagesystem 30 when the application unit 50 requests the update of the dataof a specific information system (hereinafter, referred to as “targetinformation system” in the description of the operation shown in FIG.15) stored in the data storage system 40.

As the cases where the application unit 50 requests the update of thedata of the target information system stored in the data storage system40, for example, there is a case where, in response to an instructionfrom a user of the application service of the application unit 50, thisapplication service requests the update of the data of the targetinformation system stored in the data storage system 40.

As shown in FIG. 15, the application unit 50 requests the management API65 to update the data of the target information system stored in thedata storage system 40 (S421).

When the management API 65 receives the request at S421, it notifies thepipeline orchestrator 61 of an event indicating the received request(S422).

When the trigger processing unit 81 of the pipeline orchestrator 61receives the event notified by the management API 65 at S422, thetrigger processing unit 81 analyzes the content of this event, calls ascenario corresponding to this event, that is, a scenario of the updateof the data of the target information system stored in the data storagesystem 40 from the action description unit 82 (S423), and notifies thescenario called at S423 to the action processing unit 83 (S424).Therefore, the action processing unit 83 executes the processing basedon the scenario notified at S424. That is, the action processing unit 83first confirms whether or not the data of the target information systemstored in the data storage system 40 is the latest (S425). As a resultof the confirmation at S425, if the data of the target informationsystem stored in the data storage system 40 is not the latest, theaction processing unit 83 instructs transmission of the data of thetarget information system to the data collection system for the targetinformation system (S426).

Therefore, the data collection system acquires data from the targetinformation system (S427) and passes the data acquired at S427 to thepipeline associated with the data collection system itself (S428).

After the processing at S428, the processing shown in FIGS. 10 and 11 isexecuted.

When the application unit 50 requests the update of the data of thetarget information system stored in the data storage system 40, wherebythe pipeline 70 and the big data analysis unit 44 process the data, thefinal conversion processing by the big data analysis unit 44 ispreferably completed early. Therefore, regarding the processing at S354,the data transfer processing unit 73 may transfer the data passed fromthe masking processing unit 72 at S348 directly to the big data analysisunit 44 instead of transfer of the data stored in the secondary storage74 at S353 to the big data analysis unit 44 via the secondary storage74.

In the above, the update of the data of the target information systemstored in the data storage system 40 has been described. Here, the datalinkage system 30 can also update only specific data among the data ofthe target information system stored in the data storage system 40. Forexample, the data linkage system 30 can also update only data in aspecific device management table among the data of the targetinformation system stored in the data storage system 40.

Next, the operation of the data linkage system 30 when it changes itsown configuration in response to a change in the configuration of aspecific information system will be described.

FIG. 16 is a flowchart of an operation of the data linkage system 30when it changes its own configuration in response to a change in theconfiguration of a specific information system (hereinafter, referred toas “target information system” in the description of the operation shownin FIG. 16).

The configuration management gateway 63 executes the operation shown inFIG. 16 at a specific timing.

As shown in FIG. 16, the configuration management gateway 63 connects tothe configuration management server of the target information system(S441) and determines whether or not there is a change in theconfiguration of the data to be linked on the basis of the informationfrom the configuration management server of the target informationsystem (S442).

When the configuration management gateway 63 determines at S442 thatthere is no change in the configuration of the data to be linked, theconfiguration management gateway 63 ends the operation shown in FIG. 16.

When it is determined at S442 that there is a change in theconfiguration of the data to be linked, the configuration managementserver 62 determines whether or not the content of the change in theconfiguration to be changed in response to the content of the change inthe configuration of the data to be linked among the configurations ofthe data collection system and the data storage system 40 is defined(S443). Here, the configuration management server 62 stores changecontent correspondence relationship information indicating thecorrespondence relationship between the content of the change in theconfiguration of the data to be linked and the content of the change inthe configuration to be changed in response to the content of the changein the configuration of the data to be linked among the configurationsof the data collection system and the data storage system 40. When thecorrespondence relationship regarding the content of the change in theconfiguration of the data to be linked is stored in the change contentcorrespondence relationship information, the configuration managementserver 62 determines that the content of the change in the configurationto be changed in response to the content of the change in theconfiguration of the data to be linked among the configurations of thedata collection system and the data storage system 40 is defined. On theother hand, when the correspondence relationship regarding the contentof the change in the configuration of the data to be linked is notstored in the change content correspondence relationship information,the configuration management server 62 determines that the content ofthe change in the configuration to be changed in response to the contentof the change in the configuration of the data to be linked among theconfigurations of the data collection system and the data storage system40 is not defined.

When the configuration management server 62 determines at S443 that thecontent of the change in the configuration to be changed in response tothe content of the change in the configuration of the data to be linkedamong the configurations of the data collection system and the datastorage system 40 is not defined, the configuration management server 62stops the processing of the data collection system and the data storagesystem 40 regarding the data to be linked (S444). Next, theconfiguration management server 62 informs that the configuration of thedata linkage system 30 cannot be changed in response to the change inthe configuration of the target information system to a predetermineddestination such as the destination of a person in charge of the targetinformation system, for example, (S445) and ends the operation shown inFIG. 16.

When the configuration management server 62 determines at S443 that thecontent of the change in the configuration to be changed in response tothe content of the change in the configuration of the data to be linkedamong the configurations of the data collection system and the datastorage system 40 is defined, the configuration management server 62changes the configuration to be changed in response to the content ofthe change in the configuration of the data to be linked in the datacollection system and the data storage system 40 with the content of thechange defined in the change content correspondence relationshipinformation (S446). Here, as the content of the change in theconfiguration of the data collection system, for example, a change in arange of data to be linked, a change in a frequency of linkage and thelike can be considered. When the configuration management server 62changes the configuration of the data collection system, theconfiguration management server 62 may deploy a new data collectionsystem with the changed configuration. As the content of the change inthe configuration of the data storage system 40, for example, the changein the processing content of the masking processing by the maskingprocessing unit or the change in the processing content of the finalconversion processing in the big data analysis unit 44 can beconsidered.

The configuration management server 62 ends the operation shown in FIG.16 after the processing of S446.

As described above, when processing which failed is detected among theprocessing at each stage on the data in the data linkage system 30itself (S401), the data linkage system 30 has this processingre-executed (S404) and thus, the data can be linked even if theprocessing fails during the linkage.

For example, when a failure such as a communication error with the datasource unit 20 occurs, recovery from the failure, that is, re-executionof the data collection processing is executed automatically and in aminimum range in the data linkage system 30 and thus, the operating costof the entire data linkage system 30 can be reduced even when a largeamount of data is linked.

When the application unit 50 requests the pipeline orchestrator 61 toupdate the data stored in the data storage system 40 (S421 to S422), thedata linkage system 30 has the processing of collecting the data forwhich the update was requested by the application unit 50 executed bythe data collection system (S426) and thus, the data linkage system 30can easily update the data stored in the data storage system 40.

In the present embodiment, the pipeline includes a masking processingunit as a data conversion system. However, the pipeline may include atleast one data conversion system other than the masking processing unitin place of the masking processing unit or in addition to the maskingprocessing unit.

What is claimed is:
 1. A data linkage system comprising: a datacollection system that collects data held by an information system and adata storage system that stores data held by a plurality of theinformation systems and collected by the data collection system; and aprocessing monitoring system that monitors processing at each stage,that is executed in the data linkage system on the data stored in thedata storage system, wherein, the processing monitoring system detectsfailure of the processing executed on the data and instructs a processorof the failed processing to re-execute the processing on the data, theprocessor detecting the failure of the processing and notifying theprocessing monitoring system of the failure.
 2. The data linkage systemaccording to claim 1, further comprising an application unit that usesdata stored in the data storage system, wherein, when update of the datastored in the data storage system is requested by the application unit,the processing monitoring system has the processing of collecting thedata for which the update was requested by the application unit executedby the data collection system.
 3. The data linkage system according toclaim 1, wherein the processor includes a primary storage that storesthe data collected by the data collection system and a data conversionsystem that executes a predetermined processing on target data includedin the data stored in the primary storage, and wherein the dataconversion system re-executes the predetermined processing on the targetdata stored in the primary storage in accordance with a re-executioninstruction from the data processing monitoring system.
 4. A processingmonitoring system that monitors processing at each stage, that isexecuted on data in a data linkage system which includes a datacollection system that collects data held by an information system and adata storage system that stores data held by a plurality of theinformation systems and collected by the data collection system, whereinthe data on which the processing is executed is the data stored in thedata storage system, and wherein, when the processing monitoring systemdetects failure of the processing executed on the data and instructs aprocessor of the failed processing to re-execute the processing to thedata, the processor detecting the failure of the processing andnotifying the processing monitoring system of the failure.
 5. A datalinkage system comprising: a data collection system that collects dataheld by an information system; a data storage system that stores dataheld by a plurality of the information systems and collected by the datacollection system; a processing monitoring system that monitorsprocessing at each stage on data in the data linkage system; and anapplication unit that uses data stored in the data storage system,wherein when processing which failed is detected, the processingmonitoring system has this processing re-executed, and wherein whenupdate of the data stored in the data storage system is requested by theapplication unit, the processing monitoring system has the processing ofcollecting the data for which the update was requested by theapplication unit executed by the data collection system.
 6. A processingmonitoring system that monitors processing at each stage on data in adata linkage system which includes a data collection system thatcollects data held by an information system, a data storage system thatstores data held by a plurality of the information systems and collectedby the data collection system, and an application unit that uses datastored in the data storage system, wherein when processing which failedis detected, the processing monitoring system has this processingre-executed, and wherein when update of the data stored in the datastorage system is requested by the application unit, the processingmonitoring system has the processing of collecting the data for whichthe update was requested by the application unit executed by the datacollection system.